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(54) Image processing method and apparatus, image processing system, and storage medium 



(57) In general, it is difficult for the end user to im- 
plement a simulate illustration process for an image, 
since manual setups of the process are complicated and 
require skills. Also, there is room for improvement in 
terms of image quality. In this invention, the luminance 
histogram of an original image is generated, the bright- 
ness of that image is detected based on the histogram, 



and the image undergoes a filter process corresponding 
to the detected brightness, thus easily performing the 
simulate illustration process using bright tone for the im- 
age. Especially, since a filter is set for rasterized image 
data in correspondence with the number of lines of the 
data to be processed, an effective simulate illustration 
process can be implemented. 
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Description 

BACKGROUND OF THE INVENTION 

[0001] This invention relates to an image processing 5 
method, apparatus, and system for modifying an original 
image, and a storage medium. 
[0002] As digital cameras, photoscanners, and the 
like have prevailed in recent years, a photo image can 
be easily converted into digital data. Also, since output 
apparatuses represented by ink-jet printers have at- 
tained higher image quality and lower prices, the end 
user can easily output a photo onto a recording sheet at 
home. 

[0003] Hence, with an apparatus such as a personal 
computer or the like that allows image processing, the 
end user edits an image sensed by, e.g.,. a digital cam- 
era as he or she so desires, and outputs the edit result 
via an ink-jet printer. 

[0004] A process called posterization has been pro- 
posed as Japanese Patent Laid-Open No. 1-314389, 
and an illustration edit process has been proposed as 
Japanese Patent Laid-Open Nos. 3-91088 and 
3-91087. 

[0005] As the personal computer and printer that can 
implement an edit process of a digital photo image have 
higher performance and lower prices, for example, the 
end user can easily create name cards having his or her 
own face photo superimposed on them. 
[0006] Also, a machine that can print photo stickers 
of a sensed image at that place has prevailed. 
[0007] A case will be examined below wherein a photo 
is superimposed on a name card. Upon creating formal 
name cards, it is a common practice to superimpose an 
expressionless face photo. However, as such name 
cards sometimes produce stiffness, informal name 
cards are often required. In such case, the need for su- 
perimposing not only a face photo that the user desired 
but a more personalized photo created by editing the 
photo itself has arisen. As an edit process that meets 
such need, a monochrome edit process represented by 
sepia-tone conversion, an edit process for converting a 
photo into an illustration, and the like are known. 
[0008] In this manner, various needs for photo images 
have increased recently. 

[0009] However, in the conventional image process- 
ing apparatus, in order to edit a photo image to obtain 
an illustration-style image, the edges of the photo image 
are extracted and individual portions are painted in the 
same colors as those in the original image. Furthermore, 
the user must finely set various parameters required for 
edge extraction, painting, and the like. 
[0010] Hence, in order to obtain an illustration-style 
image, room for improvement still remains in terms of 
image quality. Also, not only complicated setups but also 
considerable user skills are required to obtain appropri- 
ate setups. Hence, it is difficult for the end user to attain 
such setups. 



SUMMARY OF THE INVENTION 

[0011] Accordingly, it is a concern of the present in- 
vention to provide an image processing method, appa- 
ratus, and system which can easily perform an effective 
simulate illustration process for image data, and a stor- 
age medium. 

[0012] According the present invention, there is pro- 
vided an image processing method comprising: the 
modify step of modifying a first image to obtain a second 
image on the basis of a first signal obtained by extracting 
an edge of the first image, and a second signal obtained 
by reducing the number of tone levels. 
[0013] It is another concern of the present invention 
to provide an image" processing method, apparatus, and 
system which allows the user to easily set up details of 
simulate illustration process, and a storage medium. 
[0014] According to another aspect the present inven- 
tion, provides an image processing method further com- 
prising: the instruction input step of inputting a user in- 
struction that selects a desired one of a plurality of mod- 
ify modes; and the image process step of executing an 
image process for image data using the modify mode 
corresponding to the user instruction, and wherein the 
plurality of modify modes include an illustration mode 
for converting the first image into the second image in 
the modify step. 

[0015] It is another concern of the present invention 
to provide an image processing method, apparatus, and 
system which can easily perform an effective simulate 
illustration process-even for rasterized image data, and 
a storage medium. 

[0016] According to a further aspect .the present in- 
vention, provides an image processing method further 
comprising: the segmentation step of segmenting the 
first image into a plurality of regions: and the setting step 
of setting processing contents of each of the segmented 
regions, and wherein the modify step includes the step 
of modifying the first image in units of regions to obtain 
the second image. 

[0017] The invention is particularly advantageous 
since a simulate illustration process can be done for im- 
age data more effectively in brighter tone than a con- 
ventional method. 

[0018] Also, the user can easily set up details of sim- 
ulate illustration process. 

[0019] Furthermore, a recording medium on which an 
image that has undergone a novel process closely re- 
sembling a hand illustration is formed can be provided. 
[0020] Other features and advantages of the present 
invention will be apparent from the following description 
taken in conjunction with the accompanying drawings, 
in which like reference characters designate the same 
or similar parts throughout the figures thereof. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0021] The accompanying drawings, which are incor- 
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porated in and constitute a part of the specification, il- 
lustrate embodiments of the invention and, together with 
the description, serve to explain the principles of the in- 
vention. 

Fig. 1 is a block diagram showing the arrangement 
of an image processing system according to an em- 
bodiment of the present invention; 
Fig. 2 is a block diagram of a printer driver; 
Fig. 3 is a flow chart showing an image process in 
the printer driver; 

Fig. 4 is a flow chart showing a histogram genera- 
tion process; 

Fig. 5 shows an example of a normal filter; 

Figs. 6A to 6C show other examples of filters (5 x 

5); 

Fig. 7 shows still another example of a filter (3 x 3); 
Fig. 8 is a flow chart showing a simulate illustration 
process; 

Fig. 9 is a flow chart showing an image edit process 
according to the second embodiment of the present 
invention; 

Fig. 1 0 shows an example of a selection window for 
an image to be edited; 

Fig. 11 shows an example of a selection window for 
a modify mode; 

Fig. 12 shows an example of a setup window for 
details of an illustration mode; 
Fig. 13 is a flow chart showing an original image 
data generation process; 

Fig. 1 4 is a flow chart showing a simulate illustration 
process; 

Fig. 1 5 shows an example of a filter process for an 
image edge upon batch-processing an image; 
Figs. 16A and 16B show an example of area seg- 
mentation upon obliquely setting an image; 
Fig. 17 is a flow chart showing a simulate illustration 
process according to the third embodiment of the 
present invention; 

Figs. 1 8A and 1 8B show examples of special filters 
in the third embodiment; 

Figs. 19A to 19C show a user interface used to set 
up an image process according to the fourth em- 
bodiment of the present invention; 
Fig. 20A shows an example of an original image 
(JPEG); 

Fig. 20B shows an example of an illustration edit 
result of the original image shown in Fig. 20A using 
a 3 x 3 filter; 

Fig. 20C shows an example of an illustration edit 
result of the original image shown in Fig. 20A using 
a 5 x 5 filter; 

Fig. 20D shows an example of an illustration edit & 
monochrome edit result of the original image shown 
in Fig. 20A; 

Fig. 21 A shows an example of an original image 
(TIFF); 

Fig. 21 B shows an example of an illustration edit 



result of the original image shown in Fig. 21 A using 
a 3 x 3 fitter; and 

Fig. 21 C shows an example of an illustration edit 
result of the original image shown in Fig. 21 A using 
5 a 5 x 5 filter. 

DESCRIPTION OF THE PREFERRED 
EMBODIMENTS 

w [0022] Preferred embodiments of the present inven- 
tion will now be described in detail in accordance with 
the accompanying drawings. 

<First Embodiment 

15 

•System Arrangement 

[0023] Fig. 1 shows an example of an image process- 
ing system according to this embodiment. A printer 105 
such as an ink-jet printer or the like, and a monitor 106 
are connected to a host computer 100. 
[0024] The host computer 100 has as software appli- 
cation software programs 1 0 1 such as a wordprocessor, 
spreadsheet, Internet browser, and the like, an OS (Op- 
erating System) 102, a printer driver 103 for processing 
various drawing commands (image drawing command, 
text drawing command, graphics drawing command) in- 
dicating output images, which are issued by the appli- 
cation software programs 101 to the OS 102. and gen- 
erating print data, and a monitor driver 1 04 for process- 
ing various drawing commands issued by the applica- 
tion software programs 101 and displaying data on the 
monitor 106. 

[0025] Reference numeral 1 1 2 denotes an instruction 
input device; and 113, its device driver. For example, a 
mouse that points various kinds of information displayed 
on the monitor 106 to issue various instructions to the 
OS 102 is connected. Note that other pointing devices 
such as a trackball, pen, touch panel, and the like, or a 
keyboard may be "connected in place of the mouse. 
[0026] The host computer 100 comprises, as various 
kinds of hardware that can run these software programs, 
a central processing unit (CPU) 108. hard disk (HD) 107. 
random-access memory (RAM) 109. read-only memory 
(ROM) 110, and the like. 

[0027] As an example of the image processing sys- 
tem shown in Fig. 1, Windows98 available from Micro- 
soft Corp. is installed as an OS in a popular PC-AT com- 
patible personal computer available from IBM Corp.. de- 
sired application programs that can implement printing 
are installed, and a monitor and printer are connected 
to the personal computer. 

[0028] In the host computer 100. each application 
software program 101 generates output image data us- 
ing text data which is classified into text such as char- 
acters or the like, graphics data which is classified into 
graphics such as figures or the like, image data which 
is classified into a natural image or the like, and so forth. 
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Upon printing out the output image data, the application 
software program 101 sends a print-out request to the 
OS 102. At this time, the application software program 
1 01 issues a drawing command group which includes a 
graphics drawing command corresponding to graphics 5 
data, and an image drawing command corresponding 
to image data to the OS 102. 

[0029] Upon receiving the output request from the ap- 
plication software program 101, the OS 102 issues a 
drawing command group to the printer driver 1 03 corre- 
sponding to an output printer. The printer driver 103 
processes the print request and drawing command 
group input from the OS 102, generates print data that 
the printer 105 can print, and transfers the print data to 
the printer 105. When the printer 105 is a raster printer, 
the printer driver 103 performs an image correction 
process for the drawing commands from the OS 102, 
and then rasterizes the commands in turn on the RGB 
24-bit page memory. Upon completion of rasterization 
of all the drawing command, the printer driver 103 con- 
verts the contents of the RGB 24-bit page memory into 
a data format that the printer 105 can print, i.e., CMYK 
data, and transfers the converted data to the printer 1 05. 
[0030] Note that the host computer 100 can connect 
a digital camera 111 which senses an object image and 
generates RGB image data, and can load and store the 
sensed image data in the HD 107. Note that the image 
data sensed by the digital camera 111 is encoded by 
JPEG. The sensed image data can be transferred as 
image data to the printer 105 after it is decoded by the 
printer driver 103. 

•Printer Driver Process 

[0031] The process executed by the printer driver 103 
will be explained below with reference to Fig. 2. 
[0032] In the printer driver 103, an image correction 
processing unit 120 executes an image correction proc- 
ess and image edit process (to modify image data; to 
be described later) for color information of an image 
drawing command included in the drawing command 
group received from the OS 102. Then, a printer correc- 
tion processing unit 121 rasterizes the drawing com- 
mands in accordance with the processed color informa- 
tion to generate dot image data on the RGB 24-bit page 
memory. The unit 121 executes a masking process, 
gamma correction process, quantization process, and 
the like in units of pixels in correspondence with color 
reproducibility of the printer to generate CMYK data de- 
pending on the printer characteristics, and transfers 
them to the printer 105. 

[0033] An image process for an original image indi- 
cated by an image drawing command in the image cor- 
rection processing unit 1 20 will be explained below with 
reference to the flowchart in Fig. 3. Note that an original 
image indicated by a graphics drawing command or text 
drawing command does not undergo a process to be 
described below. Assume that an original image is 



stored in a predetermined area in the RAM 109. 
[0034] The image correction processing unit 120 of 
this embodiment decodes the original image if it is en- 
coded by, e.g., JPEG (S20, S21), as shown in Fig. 3. 
After that, the unit 1 20 executes a histogram generation 
process (S22) and an image correction process (S23) 
in accordance with the generated histogram, and then 
executes a simulate illustration process (S24) in accord- 
ance with the generated histogram. 
[0035] The image that has undergone the simulate il- 
lustration process is output to the printer 105 as printa- 
ble data via the printer correction processing unit 121, 
and is printed out onto a storage medium. 

•Generation of Luminance Histogram 

[0036] Fig. 4 is a flow chart showing the histogram 
generation process in step S22. 
[0037] Referring to Fig. 4, when the control enters an 
original image luminance histogram generation routine 
in step S1 , the selection ratio of pixels used to generate 
a luminance histogram to those of an original image is 
determined in step S2. In this embodiment, when image 
data to be processed has 350,000 pixels, a luminance 
histogram is generated for all the pixels (selection ratio 
= 1 (or 100%)). When image data having more than 
350,000 pixels is input, pixel selection (sampling) is 
done in accordance with the ratio of the total number of 
pixels to 350,000 pixels. For example, when image data 
having 3,500,000 pixels is input, the selection ratio is 
3,500,000/350,000 *= 10, and a luminance histogram is 
generated at a ratio of one pixel per 10 pixels (selection 
ratio o 10 (or 10%). The selection ratio n in this embod- 
iment is given by: 

n = int(total number of pixels of image data to be proc- 
essed/350,000 as reference number of pixels) 

(for n = 1 when n < 1, n is a positive number) 
[0038] Subsequently, a counter that manages a line 
number is reset or set at a predetermined initial value in 
step S3, and that counter is incremented to indicate the 
line number of the line of interest in step S4. 
[0039] In this embodiment, since pixel thinning (sam- 
pling) is done in units of lines, when the selection ratio 
= n, and the remainder obtained upon dividing the line 
number by n is 0, pixels which belong to that line are 
selected as those to be processed (S5 = YES). For ex- 
ample, when the selection ratio = 10, and the remainder 
obtained upon dividing the line number by 1 0 is 0, pixels 
which belong to that line are selected as those to be 
processed. 

[0040] If the line of interest is a line to be thinned out, 
i.e., a line which is not to be processed, the flow returns 
to step S4. If the line of interest is a line to be processed, 
the flow advances to step S6, each of pixels which be- 
long to the line of interest is selected in turn, and lumi- 
nance conversion and chromaticity conversion are done 
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for that pixel of interest. Luminance conversion and 
chromaticity conversion in this embodiment are attained 

by: 

Y (luminance) = int(0.30R + 0.59G + 0.11 B) 
(Y is a positive integer) 

C1 (chromaticity) = R - Y 

C2 (chromaticity) = B - Y 
Note that luminance and chromaticity conversion equa- 
tions are not limited to the above ones, but may use var- 
ious other equations. 

[0041] In this embodiment, in order to improve the de- 
tection precision of highlight and shadow points, a sat- 
uration S of the pixel of interest is calculated by: 

Saturation S = V(C1 2 + C2 2 ) 

It is then checked if the saturation S is larger than a pre- 
determined saturation value (Sconst) (S7). If S > 
Sconst, information of that pixel is not reflected in the 
luminance histogram. 

[0042] That is, if S > Sconst, the flow returns to step 
S6 not to reflect data of that pixel in the subsequent proc- 
esses. In general, since the saturation of the highlight 
point is given by the average saturation of a high-lumi- 
nance pixel group, that saturation value includes errors 
produced by color cast. Hence, a pixel which has a sat- 
uration higher than the predetermined value is prefera- 
bly excluded from a highlight point calculation. 
[0043] An example of the effect of this process will be 
explained below. For example, the luminance Y and sat- 
uration S of a yellow pixel (R = G = 255, B = 0) are re- 
spectively 226 and 227 from the above equations. That 
is. this pixel has a very high luminance and assumes a 
sufficiently high saturation color. It is appropriate to de- 
termine that such pixel is an original yellow pixel rather 
than to determine that the pixel is obtained as a result 
of yellow cast of an achromatic pixel. When a luminance 
histogram including such high-luminance/high-satura- 
tion pixels is generated, errors are produced in the high- 
light point detection result. Hence, in this embodiment, 
the predetermined saturation (Sconst) is determined, 
and pixels having saturation beyond this value are ex- 
cluded from the luminance histogram. In this manner, 
the highlight point detection result can be prevented 
from suffering any errors due to such high-saturation 
pixels, and the detection precision of the highlight point 
can be improved. 

[0044] However, if the average density of an image 
need only be detected, as will be described later, the 
aforementioned process in step S7 in this embodiment 
may be omitted. 

[0045] After the decision in step S7, a luminance his- 
togram is generated using pixels that satisfy a condition 
(S < Sconst) (S8). Since pixel data RGB to be processed 
in this embodiment are 8-bit (256 tone levels) data, the 
luminance Y is also converted into a depth of 256. 
Hence, the luminance histogram is obtained by counting 



the frequency of occurrence of pixels corresponding to 
each of 256-level luminance values ranging from 0 to 
255. 

[0046] Note that the calculated chromaticity values 
5 C1 and C2 can be used as data for calculating the av- 
erage chromaticity of pixels having individual luminance 
values in, e.g., a color cast correction process. Hence, 
in this embodiment, data are held as follows. Three 
members, i.e., the frequency of occurrence, C1 accu- 
se mulated value, andC2 accumulated value, are set in the 
format of a structure array variable which has an index 
range from 0 to 255, and the computation result of each 
pixel is reflected in each member having the luminance 
value of that pixel as an index. 
15 [0047] Upon completion of processing for the pixel of 
interest, it is checked if ail the pixels in the line of interest 
have been processed (S9). If pixels to be processed still 
remain in the line of interest, the flow returns to step S6 
to repeat the processes in step S6 and the subsequent 
20 steps. Upon completion of processing for all the pixels 
in the line of interest, it is checked in step S10 if lines to 
be processed still remain. If processing for all the lines 
is complete, the flow ends in step S11; otherwise, the 
flow returns to step S4 to select the next line as the line 
25 of interest, thus repeating the aforementioned process- 
es. 

[0048] In this manner, since the luminance histogram 
is generated while selecting pixels of original image da- 
ta, it can be generated using a minimum required 
30 number of pixels in consideration of improvement of the 
precision upon detecting highlight and shadow points. 

•Image Correction Process 

35 [0049] In step S23 in Fig. 3, the original image under- 
goes an image correction process based on the lumi- 
nance histogram obtained in step S22. For example, 
■highlight and shadow points of the original image are 
• detected based on the luminance histogram,' and image 
40 correction processes are made based on the obtained 
highlight and shadow points. More specifically, in the 
color correction processes, color cast correction for cor- 
recting color cast of the original image, exposure cor- 
rection for correcting the contrast of luminance to opti- 
cs mize the exposure state of the original image, saturation 
correction for improving the appearance of colors of an 
output image, and the like are done. Note that these im- 
age correction processes can use known methods, and 
a detailed description thereof will be omitted. Note that 
50 the correction processes described in step S23 may be 
omitted in this invention. 

•Simulate illustration Process 

55 [0050] In step S24 in Fig. 3, the original image cor- 
rected in step S23 undergoes a simulate illustration 
process based on the luminance histogram obtained in 
step S22. This embodiment is characterized in that the 
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original image sensed by, e.g., the digital camera 111 
undergoes the simulate illustration process for convert- 
ing the original image into a hand-illustrated like image. 
[0051] The principle of the simulate illustration proc- 
ess in this embodiment will be explained first. In this em- 
bodiment, the original image is filtered using a 5 pixels 
:< 5 pixels (to be simply referred to as 5 x 5 hereinafter) 
filter that satisfies a predetermined condition with re- 
spect to the original image to be converted into an image 
in which edges are extracted (emphasized) and color 
tone is saved. Fig. 5 shows an example of this filter. For 
example, when an original image shown in Fig. 20A is 
filtered using a filter AO shown in Fig. 5, an image shown 
in Fig. 20C is obtained. As can be seen from Fig. 20C, 
since the edges of the original image as a photo image 
are extracted and the number of tone levels decreases 
and the brightness of the image increases, the original 
image is converted into an image that gives an impres- 
sion closely resembling a hand illustration. Note that the 
image shown in Fig. 20A is sensed by PowerShotAS 
(tradename) as a digital camera available from CANON 
Corp., and has 810,000 pixels. 
[0052] A filter that implements the simulate illustration 
process of this embodiment will be explained below. 
[0053] As a filter for extracting edges of an image, for 
example, a Laplacian filter is known. In general, in the 
Laplacian filter, the coefficient (weight) of the pixel of in- 
terest located at the center of the filter is set to be larger 
than those of surrounding pixels to allow extraction of a 
density change point in the image, i.e., an edge. Nor- 
mally, the sum total of coefficients in the Laplacian filter 
for edge extraction is "0". 

[0054] The filter for the simulate illustration process 
(to be simply referred to as a "filter" hereinafter) in this 
embodiment preferably uses, e.g., the filter 40 shown in 
Fig. 5, as described above, i.e., a 5 x 5 filter consisting 
of coefficients shown in Fig. 5. In the filter 40, since the 
coefficient of the pixel of interest is set at "26", and co- 
efficients of all surrounding pixels are set at "-1 ", the sum 
total of coefficients is "2". 

[0055] The filter in this embodiment is not limited to 
the filter 40 shown in Fig. 5, and the sum total of coeffi- 
cients need only be equal to or larger than zero. Figs. 
6A to 6C show other examples of the filter. Fig. 6A shows 
a filter 61 in which the coefficient of the pixel of interest 
is set at "25" by subtracting "1" from that of the filter 40, 
and Fig. 6B shows a filter 62 in which the coefficient of 
the pixel of interest is set at "27" by adding "1 " to that of 
the filter 40. An image filtered using the filter 61 becomes 
darker than that obtained by the filter 40, and an image 
filtered using the filter 62 becomes brighter than that ob- 
tained by the filter 40. Hence, according to the charac- 
teristics of these filtering results, if the filter 40 is used 
as a normal filter, the filters 61 and 62 are respectively 
used as relatively dark and bright filters. Such filter char- 
acteristics can be easily understood from the following 
fact. That is, assuming that a uniform density area of an 
image is filtered, if the sum total of filter coefficients is 



"2", the signal value, i.e., its brightness is doubled; if the 
sum total of filter coefficients is "3", its brightness is tri- 
pled. 

[0056] Fig. 6C shows a filter 63 in which coefficients 

5 of pixels around the pixel of interest in the filter 40 are 
thinned out. In this filter, the coefficient of the pixel of 
interest is set at "18" to maintain the sum total of coef- 
ficients to be "2". According to the filtering process using 
the filter 63, since the number of coefficients decreases, 

w the computation volume upon filtering can be reduced, 
and the processing speed can be improved. Note that 
the coefficients in the filter 62 are thinned out in consid- 
eration of edge detection in the vertical, horizontal, and 
oblique directions. 

15 [0057] The 5x5 filters have been explained as ex- 
amples of the filters used in this embodiment. This em- 
bodiment can also be practiced using filters having'other 
sizes. For example, the filter size can be defined by n v 
n (n = 2 x m + 1 (m is an integer equal to or larger than 

20 1)). Also, the filter may have different vertical and hori- 
zontal sizes. Fig. 7 shows a 3 x 3 filter as an example 
of the filter having another size. When an original image 
shown in Fig. 20A is filtered using the 3 • 3 filter shown 
in Fig. 7, an image shown in Fig. 20B is obtained. In the 

25 image shown in Fig. 20B as well, edges are extracted 
from the original image as a photo image and the 
number of tone levels decreases, and the brightness in- 
creases. However, the illustration edit effect is lower 
than that of the image shown in Fig. 20C based on the 

30 5 x 5 filter 40. This is because the 5 ■ 5 filter acts to 
express the extracted edges thicker. 
[0058] In the simulate illustration process in this em- 
bodiment, the thickness of the extracted edges in the 
converted image changes depending on the filter size. 

35 Hence, an optimal filter size upon simulate illustration 
process is not limited to 5 x 5 or 3 • 3 described in this 
embodiment, and depends on the image size or resolu- 
tion of an original image, fineness of an object in an im- 
age, and the like. For example, when a large filter is ap- 

^o plied to a large-size or high-resolution image or an im- 
age including a coarse object, or when a small filter is 
applied to a small-size or low-resolution image or an im- 
age including a fine object, the extracted edges become 
thicker in the former case or thinner in the latter case. 

45 On the other hand, the filter size may be set in corre- 
spondence with the brightness of an object. Further- 
more, the filter size may be set in correspondence with 
the processing speed required. 
[0059] In the simulate illustration process in this em- 

50 bodiment, the number of tone levels of image data is 
decreased, but the brightness need not always be in- 
creased. 

[0060] Such filter process may be automatically se- 
lected in correspondence with the size or resolution of 
55 an image or the fineness of an object, or may be select- 
ed in accordance with a manual instruction of the user. 
Hence, the present invention includes provision of a us- 
er interface for manual selection. 



6 



11 



EP1 014 688 A2 



12 



[0061] In step S24 in Fig. 3, the original image cor- 
rected in step S23 undergoes the simulate illustration 
process using the aforementioned filter. Fig. 8 is a flow 
chart showing the simulate illustration process in this 
embodiment. In step S81 , the brightness of an image is 
detected on the basis of the luminance histogram gen- 
erated in step S22 above. As the detection method, the 
average value, median, or maximum frequency value 
may be calculated based on the luminance histogram, 
and is compared with a predetermined threshold value 
to detect a plurality of brightness levels. For example, 
by setting a predetermined range corresponding to a 
"normal" brightness level in advance, an image which 
has brightness higher than that predetermined range is 
determined to have "bright" level, and an image which 
has brightness lower than the predetermined range is 
determined to have "dark" level, thus allowing detection 
of three different brightness levels. 
[0062] Based on the brightness level detected in step 
S81, a filter is set in step S82, S83, or S84. That is, if 
the original image has "bright" level, the filter 61 as a 
relatively dark filter is set in step S82 to decrease the 
brightness of the conversion result; if the original image 
has "normal" level, the filter 40 as a normal filter is set 
in step S83; or if the original image has "dark" level, the 
filter 62 as a relatively bright filter is set in step S84 to 
increase the brightness of the converted image. 
[0063] In step S85, the original image is filtered using 
the selected filter, thus implementing appropriate simu- 
late illustration process. 

[0064] Note that the number of brightness levels to be 
determined is not particularly limited, and can be set in 
correspondence with filters that can provide appropriate 
effects in units of levels. 

[0065] In this embodiment, since the printer driver 1 05 
executes the simulate illustration process, the entire re- 
gion of the original image cannot undergo a batch proc- 
ess. For this reason, the original image is segmented 
into region blocks each consisting of a plurality of lines, 
and the illustration process is done in units of blocks. 
Hence, in order to avoid block boundaries from becom- 
ing conspicuous in the converted image, the filter size 
is changed to a smaller one, e.g., 3 x 5 or the like near 
the final line of each block. In this manner, continuity of 
block boundaries can be maintained in the converted 
image. 

[0066] As described above, according to this embod- 
iment, an original image can easily undergo simulate il- 
lustration process only with modifying image data, and 
a hand-illustrated like image can be obtained with an 
atmosphere of the original image, since its lightness in- 
creases. As the illustration-edited image has a smaller 
number of tone levels than the original image, the data 
size of the image can be reduced. 
[0067] In this embodiment, the luminance histogram 
of an original image is generated, and the original image 
undergoes the image correction process based on that 
luminance histogram. After that the simulate illustration 



process is executed based on the luminance histogram. 
However, the image correction process based on the lu- 
minance histogram need not always be required. When 
the simulate illustration process alone is done immedi- 
5 ately after the luminance histogram is generated, the 
processing speed can be improved. 
[0068] When the user manually designates bright- 
ness he or she wants in correspondence with the expo- 
sure state or the like of an original image, the histogram 
generation process in step S22 may be omitted. 
[0069] On the other hand, filter control in the simulate 
illustration process may be done on the basis of param- 
eters based on the image correction process such as 
contrast information between the background and ob- 
is ject, scene information to which an image belongs, and 
the like, which are used as discrimination conditions up- 
on executing color correction or the like in the image cor- 
rection process. 

[0070] In this embodiment, a histogram of a lumi- 

20 nance signal Y is generated. Alternatively, brightness 
detection may be done based on the histogram of G 
components without executing any luminance/chroma- 
ticity conversion of original image data. In this case, an 
increase in processing speed can be expected since no 

25 luminance/chromaticity conversion is done, 

[0071] In this embodiment, an original image is RGB 
data. Of course, the present invention can be applied 
even when the original image has other formats. For ex- 
ample, original image data having another format may 

30 be converted into the RGB format, and the individual 
color components of the converted data may undergo 
the simulate illustration process of this embodiment or 
a filter corresponding to the data format of the original 
image may be applied. For example, when an original 

35 image has a YMC format, the signs of the coefficients 
of the filters used in this embodiment may be inverted. 
Even when an original image has a YHS format or the 
like, the present invention can be applied. For example, 
when an original image has a format such as an Lab 

^0 format or the like that allows extraction of only a lumi- 
nance component, only the luminance component can 
be filtered, thus reducing the computation volume com- 
pared to that of filtering in units of components of the 
RGB format. 

45 [0072] The image to be subjected to simulate illustra- 
tion process in this embodiment is not limited to an im- 
age sensed by the digital camera. For example, a photo 
image scanned by, e.g., a photoscanner or the like may 
be input, or a photo image stored in a medium of an ex- 

50 ternal device such as a CD-ROM may be used. 

[0073] In this embodiment, the filter coefficients are 
controlled to implement the simulate illustration proc- 
ess. However, an illustration-edited image obtained by 
extracting image edges and by decreasing the number 

55 of tone levels and by increasing its brightness can also 
be obtained by a method other than filter coefficient con- 
trol. For example, a program may be set in advance to 
automatically execute a process for increasing the 
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brightness of the entire image and decreasing the 
number of tone levels, after an original image undergoes 
a conversion process such as an oil paint process or the 
like to obtain a hand-painted like image by an application 
such as Photoshop (tradename) available from Adobe 
Corp., thus obtaining a hand-illustrated like image 
shown in Fig. 20C. 

<Second Embodiment 

[0074] The second embodiment of the present inven- 
tion will be described below. 

[0075] In the first embodiment, the simulate illustra- 
tion process is automatically executed for an original im- 
age. In the second embodiment, an image processing 
system that allows the user to input an arbitrary edit in- 
struction will be explained. 

[0076] Since the system arrangement in the second 
embodiment is the same as that in the first embodiment, 
a detailed description thereof will be omitted. 
[0077] Fig. 9 is a flow chart showing the image edit 
process in the second embodiment. In Fig. 9, the same 
step numbers denote the same processes as those in 
Fig. 3 in the first embodiment. Note that of the processes 
shown in Fig. 9, processes which require a user inter- 
face are implemented by the predetermined application 
software programs 101 , and other processes are imple- 
mented by the image correction processing unit 120 in 
the printer driver 1 03. 

•Target Image Selection 

[0078] In step S31, the user selects an image to be 
edited. Fig. 10 shows an example of the selection win- 
dow for an image to be edited, which is displayed on the 
monitor 106 upon launching a predetermined image ap- 
plication 101 . Referring to Fig. 10, reference numeral 11 
denotes an image display frame for displaying a list of 
reduced-scale images (thumbnail images) of candidate 
images which may undergo the edit process of this em- 
bodiment. These images may be input from the HD 1 07 
or a medium of an external storage device such as a 
CD-ROM or the like, or may be read out from the RAM 
109 or ROM 110. The images displayed on the image 
display frame 11 are not limited to thumbnail images. 
For example, an arbitrary one of thumbnail images dis- 
played in a list may be selected, and may be displayed 
after it is decoded. Reference numeral 1 4 denotes a but- 
ton for instructing to start an image edit process; and 1 5, 
a button for instructing to end the image edit process. 
Other function buttons may be provided. Note that text 
display "list of images" displayed above the "edit" button 
14 indicates that the current display on the image dis- 
play frame 11 is list display of images. 
[0079] On the target image selection window shown 
in Fig. 10, the user moves a cursor 16 on the screen 
onto a desired image on the image display frame 11 by 
the instruction input device such as a keyboard, mouse, 



or the like, and makes execution operation such as click- 
ing or the like, thus selecting that image. In Fig. 10, the 
image displayed on the right end of the image display 
frame is selected, and is highlighted by a bold frame. 

5 Note that a plurality of images may be selected. When 
the user makes execution operation by moving the cur- 
sor 1 6 onto the "edit" button 1 4 while at least one image 
is selected, the selected image is determined, and the 
start instruction of the image edit process is issued. The 

io selected image data is loaded onto a predetermined ar- 
ea on the RAM 109 in the RGB format. 

■Edit Mode Selection 

15 [0080] In step S32, the user then designates an edit 
mode (illustration mode in this embodiment) to be exe- 
cuted. In this embodiment, the user designates an arbi- 
trary one of a plurality of edit modes prepared in ad- 
vance. An edit mode selection process will be explained 

20 below with reference to Fig. 11 . 

[0081] Fig. 11 shows the edit mode selection window, 
which is displayed on the monitor 106 after the image 
to be edited is selected, as described above. Referring 
to Fig. 11, reference numeral 21 denotes an edit mode 

25 display frame for displaying a list of edit modes, which 
can be executed in this embodiment, together with im- 
ages that represent their outlines. The edit mode display 
frame 21 can display the previous or next page upon 
operation of a button 22 or 23, and can display other edit 

30 modes. Reference numeral 27 denotes a button 
pressed upon setting details. Reference numeral 24 de- 
notes a button for instructing to start the image edit proc- 
ess; and 25, a button used to return to the target image 
selection window shown in Fig. 10. In Fig. 11 as well, 

35 other function buttons may be provided. Note that text 
display "list of edit modes" displayed above the "return" 
button 25 indicates that the current display on the edit 
mode display frame 21 is list display of edit modes. 
[0082] A description of the detailed contents of the in- 

40 dividual edit modes will be omitted. For example, for edit 
modes (brick mode and the like) for printing predeter- 
mined texture data to overlap an original image, RGB 
data indicating a predetermined color suitable for the 
mode is stored on a predetermined area on the RAM 

45 1 09 or ROM 1 1 0 as texture data together with a prede- 
termined pattern. 

[0083] Note that the images that represent outlines of 
the individual edit modes displayed on the edit mode dis- 
play frame 21 in Fig. 11 may be prepared in advance in 

so the apparatus. For example , the image data to be edited 
selected in Fig. 10 or its thumbnail data may actually 
undergo edit processes corresponding to the individual 
modes, and the process results may be displayed (pre- 
viewed) on the edit mode display frame 21 . 

55 [0084] On the edit mode selection window shown in 
Fig. 11 , the user moves a cursor 26 on the screen onto 
a desired edit mode on the edit mode display frame 21 
by the instruction input device such as a keyboard, 
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mouse, or the like, and makes execution operation such 
as clicking or the like, thus selecting that edit mode. In 
Fig. 11, an "illustration" mode is selected and is high- 
lighted by a bold frame. 

•Detail setup for Simulate illustration process 

[0085] When the user further clicks the detail setup 
button 27 in this state, he or she can arbitrarily set de- 
tailed parameters upon an edit process in the illustration 
mode, i.e., in the simulate illustration process. Fig. 12 
shows an example of the detail setup window. Referring 
to Fig. 12, reference numeral 31 denotes an image dis- 
play frame for displaying items that the user can set. Up- 
on operation of a button 32 or 33, the image display 
frame 31 can display other items that the user can set. 
Reference numeral 35 denotes a button used to return 
to the edit mode selection window shown in Fig. 1 1 . Note 
that text display "illustration mode details" indicates that 
the current display on the image display frame 31 is to 
set up detailed items in the simulate illustration process. 
[0086] Fields 38 and 39 displayed within the image 
display frame 3 1 are respectively used to brightness set- 
ting and grainy setting parameters. In the brightness set- 
ting field 38, the user selects a "high brightness setting" 
or "low brightness setting" button depending on whether 
he or she wants to make the illustration-edited image 
relatively bright or dark. Of course, a "normal setting- 
button is selected as a default. A filter is selected in cor- 
respondence with the selected brightness parameter. In 
the grainy setting field 39, whether or not a grainy edit 
process is executed is set by either an ON or OFF but- 
ton. For example, the ON button is selected as a default. 
[0087] Note that setup items shown in Fig. 12 are 
merely examples, and setup items based on other pa- 
rameters may be provided. For example, parameters 
that pertain to setups of filter sizes described in the first 
embodiment may be set by the user. 
[0088] Upon completion of setups of various detailed 
items by the user, the user presses the "return" button 
35 to display the edit mode selection window shown in 
Fig. 11 again. Upon clicking the "edit" button 24, the edit 
mode setup process in step S32 ends, and the simulate 
illustration process, which has been selected and set up 
in detail by the user, is started. 

[0089] The grainy edit setups in the second embodi- 
ment will be explained below. In the first embodiment 
described above, an image sensed by the digital camera 
111 mainly undergoes simulate illustration process. 
Hence, since the image to be edited has already under- 
gone block encoding, e.g., JPEG encoding, block dis- 
tortion unique to block encoding is produced although it 
is not so conspicuous. The original image example 
shown in Fig. 20A in the first embodiment is an image 
which is JPEG-encoded and then decoded. Hence, in 
the illustration-edited image shown in Fig. 20C, since 
block distortion is emphasized by filtering, fine noise 
components are produced and form grains. However, 



since grains form, the illustration-edited image becomes 
a unique one that does not give any monotonous im- 
pression. In the second embodiment, such granularity 
is used as one (grainy effect) of illustration effects, and 
5 the user can select whether or not the effect (grainy ef- 
fect) is added. 

[0090] In step S33 in Fig. 9, the image to be edited 
selected in step S31 is read out on the basis of the edit 
mode and detailed parameters set in step S32, and is 
10 . stored on a predetermined area in the RAM 1 09 as orig- 
inal image data to be edited. 

•Original Image Data Generation Process 

'5 [0091] A process executed when the illustration mode 
is selected will be explained below as an example of an 
original image data generation process in step S33. Fig. 
13 is a flow chart showing the original image data gen- 
eration process in the illustration mode. 

20 [0092] It is checked in step S41 if the grainy edit setup 
parameter is "ON". If the grainy edit setup parameter is 
"ON", i.e., the user requests to add the grainy effect, the 
flow advances to step S42 to check if the selected image 
to be edited is an image encoded by block encoding. If 

25 the image to be edited is a block-encoded image, that 
image is decoded in step S44. On the other hand, if the 
image to be edited is not a block-encoded image, the 
image is encoded by JPEG in step S43. 
[0093] In step S43, the image data to be edited is con- 

30 verted into JPEG codes irrespective of its data format. 
For example, when an image has been encoded by an- 
other format, it is decoded and is then encoded by JPEG 
to give block distortion. After that, the encoded image is 
decoded in step S44. 

35 [0094] On the other hand, if it is determined in step 
S41 that the grainy edit setup parameter is "OFF", since 
the user requested to turn off the grainy effect, the flow 
advances to step S45 to check if the selected image to 
be edited is an image encoded by JPEG. If the image 

^0 to be edited is a JPEG-encoded image, that image is 
decoded in step S46. Furthermore, the decoded image 
is smoothed in step S47 to remove block distortion. On 
the other hand, if it is determined in step S45 that the 
image to be edited is not a JPEG-encoded image, the 

45 image undergoes, e.g., a decoding process correspond- 
ing to its format, thus ending the process. 
[0095] As can be seen from the flow chart shown in 
Fig. 13, when the grainy edit setup parameter is "ON", 
the processing speed for a JPEG image becomes high- 

so er since smoothing in step S47 is skipped. 

•Image Edit (Simulate illustration) Process 

[0096] In step S34, the image edit process in the edit 
55 mode set in step S32 is executed for the original image 
data. 

[0097] A simulate illustration process as an example 
of the image edit process in step S34 will be explained 
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below. Fig. 14 is a flow chart showing the simulate illus- 
tration process. The same step numbers in Fig. 14 de- 
note the same processes as those in Fig. 8 described 
in the first embodiment. Referring to Fig. 14, a filter is 
set in step S82. S83, or S84 in correspondence with the 
contents of the brightness setup parameter in the illus- 
tration mode, which are set as described above, and are 
discriminated in step S81 . More specifically, if a "darker 
edit" setup parameter is selected, the filter 61 as a rel- 
atively dark filter is set in step S82; if a "normal edit" 
setup parameter is selected, the filter 40 as a normal 
filter is set in step S83; or if a "brighter edit" setup pa- 
rameter is selected, the filter 62 as a relatively bright 
filter is set in step S84. Based on the selected filter, the 
original image is filtered in step S85, thus implementing 
simulate illustration process that the user wanted. 
[0098] As described above, according to the second 
embodiment, the user can select whether the grainy edit 
setup parameter is ON or OFF Figs. 21 A to 21 C show 
examples of the illustration edit result when the grainy 
edit setup parameter is OFF. Fig. 21 A shows an image 
obtained by decompressing an image which has been 
compressed in the TIFF format. That is, this image does 
not have any block distortion unique to JPEG. Fig. 21 A 
shows a standard image (SHIPP) of The Institute of Im- 
age Electronics Engineers of Japan, which has 4,096 x 
3.027 (around 12.5 million) pixels. Figs. 21 B and 21 C 
respectively show the illustration edit results of the im- 
age shown in Fig. 21 A using 3x3 and 5 x 5 filters. For 
example, as can be seen from Fig. 21 C, a monochrome 
illustration edit result in which production of granularity 
is suppressed is obtained compared to the grainy image 
shown in Fig. 20C. As in Figs. 20A to 20C, the image 
filtered by the 5 x 5 filter shown in Fig. 21 C presents 
the higher illustration edit effect than the image filtered 
using the 3 x 3 filter shown in Fig. 21 B. 
[0099] As described above, according to the second 
embodiment, since the user can select an arbitrary edit 
mode and can setup the details of the selected mode, 
simulate illustration process can be achieved in the 
mode that the user wanted especially in the process. 
Furthermore, since the histogram generation process 
and the like in the first embodiment need not be execut- 
ed, high-speed processing can be realized. 
[0100] Note that the second embodiment has exem- 
plified the edit process especially in the illustration mode 
of a plurality of selectable edit modes. However, the 
present invention is not limited to such specific edit 
mode, and when a plurality of edit processes including 
simulate illustration process are executed in a predeter- 
mined order, a more effective edit result can be ob- 
tained. 

[0101] In the above embodiments, the image edit 
function is provided to the printer driver. Of course, this 
function can be implemented when the OS executes an 
application. 

[01 02] Also, in the above description, various kinds of 
information and areas required in the simulate illustra- 



tion process are assured on a predetermined area on 
the RAM 19. The areas may be assured upon launching 
an application that implements the edit process, or may 
be assured when they are needed. According to the lat- 
5 ter method, since an excessive memory area need not 
be assured, a larger memory area can be assigned to 
other processes, thus improving the memory efficiency. 

<Third Embodiment 

w 

[0103] The third embodiment of the present invention 
will be described below. 

[0104] According to the first and second embodi- 
ments, a simple simulate illustration process is imple- 
15 mented for a photo image by filtering. 

[0105] However, when this simulate illustration proc- 
ess is implemented by a printer driver, the following 
problem is posed. 

[0106] More specifically, it is a common practice for 

20 serial printers represented by ink-jet printers and the like 
to execute a raster process to save the memory size 
used. In this case, since an image is segmented and 
processed, one processing unit does not always have 
data for a plurality of lines, and data for only one line 

25 may be processed in an extreme case. Hence, when an 
image undergoes simulate illustration process by 
means of filtering, the number of pixels to be filtered oth- 
er than the pixel of interest often becomes very small. 
This means deterioration of the quality of an image ob- 

30 tained as an edit result. 

[0107] To solve such problem, the third embodiment 
will exemplify an effective simulate illustration process 
even for rasterized image data. 
[0108] Since the system arrangement and an outline 

55 of processes by the printer driver in the third embodi- 
ment are the same as those shown in Figs. 1 to 3 in the 
first embodiment, a detailed description thereof will be 
omitted. 

40 -Simulate illustration Process 

[0109] In the third embodiment as well, by filtering R, 
G, and B planes of an original image using the filter de- 
scribed in the first embodiment, the original image can 

45 be easily converted into a hand-illustrated like image. In 
the third embodiment, in order to reduce the computa- 
tion volume, RGB data are decomposed into luminance 
and color difference signals, only the luminance signal 
is filtered, and the color difference signals are multiplied 

so by a given constant in correspondence with desired 
brightness. 

[0110] Luminance (Y)/chromaticity (C1, C2) conver- 
sion are done by: 

Y = 0.30R + 0.59G + 0.11B) 
55 C1 = R-Y 

C2 = B - Y 

Note that the luminance and chromaticity conversion 
equations are not limited to the above ones, but may 
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use various other equations. 

[0111] In the third embodiment, the luminance signal 
Y undergoes, e.g., a simulate illustration process using 
the filter 40 shown in Fig. 5. Note that the coefficient 
(weight) of the pixel of interest can be set at "26" if the 5 
original image has sufficient exposure (standard bright- 
ness), i.e., need only comply with a rule that the sum 
total of coefficients that define the filter is "2". On the 
other hand, if a relatively dark image is to be obtained, 
the coefficient of the pixel of interest can be set at "25"; 
if a relatively bright image is to be obtained, the coeffi- 
cient (weight) of the pixel of interest can be set at "27", 
"28" When exposure is inappropriate, the original im- 
age may be corrected by the image correction process- 
ing unit 120. 

[0112] In this case, the user selects a desired bright- 
ness level using a user interface shown in, e.g., Fig. 
19B, and the coefficient of the pixel of interest is set in 
accordance with the selected brightness level. For ex- 
ample, when the user wants to set a standard brightness 
level, the luminance signal Y is filtered using the filter 
40 to obtain Y\ and the color difference signals C1 and 
C2 are edited by calculating: 

CV = K x C1 



C2' = K x C2 

where K is a constant corresponding to the brightness 
level designated by the user. For example, when the 
standard brightness level is set, K = 2; when a brighter 
mode is selected, K = 3, 4,... may be set. K is not limited 
to an integer. 

[011 3] The obtained Y\ C1 ', and C2' are re-converted 
into R, G, and B values, thus completing the simulate 
illustration process of the third embodiment. 

•Measure Against Rasterized Image 

[0114] However, the aforementioned problem is 
posed. That is, since the serial printer performs so- 
called rasterization unlike application software that can 
batch-process a single image, an image is segmented 
and processed and, hence, a 5 x 5 filter process may 
be disabled. 

[0115] Even when an image can be batch-processed, 
the upper, lower, right, and left end regions of an image 
cannot undergo the 5x5 filter process. A case will be 
exemplified below with reference to Fig. 15 wherein the 
5 x 5 filter process is disabled. Referring to Fig. 1 5, ref- 
erence numeral 700 denotes an image to be processed, 
in which one rectangle indicates one pixel. Assuming 
that a pixel 701 of interest located in the upper left region 
of the image 700 is to be processed, it cannot undergo, 
e.g., a 5 x 5 filter process. Therefore, the image quality 
of this region becomes different from that of other re- 



gions that can undergo the 5 x 5 filter process. Such 
problem occurs in any filter processes. However, since 
this problem normally occurs only in end portions of an 
image, it is not visually conspicuous, and is not so seri- 
ous. 

[0116] When such pixel 701 of interest is to be proc- 
essed, a method of adjusting the coefficient of the pixel 
701 of interest in accordance with the number of pixels 
(to be referred to as effective pixels hereinafter) which 
are located with a region 702 near the pixel of interest 
and allow 5x5 filtering is effective. More specifically, 
as a special filter for the pixel 701 of interest, if an output 
with a standard brightness level is required, a filter hav- 
ing a sum total of coefficients = "2" can be generated 
complying with a rule that the sum total of filter coeffi- 
cients is positive, and can be used in filtering for the re- 
gion 702. Reference numeral 703 denotes an example 
of such special filter. 

[0117] In any case, when an image is batch-proc- 
essed, a filter process can be done based on three or 
more lines or columns of pixels. In addition, since a spe- 
cial filter process is done for only image end portions, 
the filter process, i.e., the simulate illustration process 
can be completed hardly forming any visually unnatural 
regions. It is empirically known that hardly any visually 
unnatural region is formed as a result of a filter process 
for three or more lines or columns of effective pixels. 
[0118] On the other hand, when image data is raster- 
ized like in the printer driver, image data for a plurality 
of lines (e.g., three lines or more) are not always assured 
as a segmented processing unit. Also, since a single im- 
age is processed in units of a plurality of regions, if dif- 
ferent filters are used every processing units, image 
quality differences become conspicuous at the bound- 
ary lines of regions. 

[0119] In an extreme case, for example, when an im- 
age is obliquely set, as shown in Fig. 16A, the image is 
segmented into regions, as shown in Fig. 16B. In this 
case, a region 801 that can assure a rectangular region 
can undergo a normal filter process for a plurality of 
lines, but a non-rectangular region 802 must undergo a 
filter process in units of lines in the worst case. Conse- 
quently, processing result differences become noticea- 
ble in units of regions. Such problem is similarly posed 
when an image is deformed into a special shape, e.g., 
a heart shape. 

[0120] Since the principle of the simulate illustration 
process in the present invention is to emphasize the dif- 
ference between the pixel of interest and its neighboring 
pixels, a portion having an abrupt change between pix- 
els, i.e., an edge portion is emphasized. That is, since 
the essence of the present invention is to more empha- 
size the difference the pixel of interest and its surround- 
ing pixels, the aforementioned shortcomings can be 
eliminated by adjusting the coefficients in correspond- 
ence with the filter size that can be formed. 
[0121] Fig. 17 is a flow chart showing the simulate il- 
lustration process for rasterized image data in the third 
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embodiment. 

[01 22] Image data for one or a plurality of lines, which 
has been segmented into a given processing unit by ras- 
terization, is input (S131). A filter is set in correspond- 
ence with the number of lines of that image data as the 
processing unit. 

[0123] It is checked if the number of lines that form 
the data to be processed allows a normal filter process 
(S132). If the normal filter process cannot be done, the 
flow advances to step S134 to set a special filter. For 
example, when the data to be processed consists of one 
line (or one column), a special filter shown in Fig. 18A 
is used; when the data to be processed consists of two 
lines (or two columns), a special filter shown in Fig. 18B 
is used. A coefficient P of the pixel of interest in each of 
the special filters shown in Figs. 18A and 18B can be 
set in correspondence with the brightness of an image 
that the user desires as an edit result. That is, when 
standard brightness is required, the filter is set so that 
the sum total of the coefficients is "2". Even when the 
image data to be processed consists of one column of 
pixels in place of one row of them, such filter can be 
used. 

[0124] On the other hand, if the number of lines that 
form the image data to be processed allows the normal 
filter process, a normal filter (5x5 filter in this embod- 
iment) is set in step S133. 

[0125] In step S135, the data to be processed is fil- 
tered on the basis of the filter set in step S133 or S134, 
thus implementing simulate illustration process in the 
third embodiment. The processes in steps S1 31 to S1 35 
are repeated until it is determined in step S136 that the 
simulate illustration process is complete for all the lines 
of image data. 

[01 26] Note that the special filters in the third embod- 
iment are not limited to the examples shown in Figs. 18A 
and 18B, but may have any other pxq sizes (p and q 
are integers equal to or larger than 1 ). 
[0127] As can be seen from Figs. 18A and 18B, sur- 
rounding pixels other than the pixel of interest are set to 
have larger coefficients with decreasing special filter 
size. The coefficient of the pixel of interest is determined 
based on those coefficients of the surrounding pixels. 
[0128] Note that the coefficients of the surrounding 
pixel in the special filter are preferably set to be the pow- 
ers of 2. In this manner, shift operations are allowed, 
thus expecting a light arithmetic load and a short 
processing time. 

[0129] Note that the number of "effective pixels" may 
become short even for image data having a number of 
lines that allow the normal filter process. In such case, 
by similarly applying the aforementioned principle, any 
problems on the boundary lines of segmented regions 
can be avoided. 

[01 30] As larger coefficients are set, features (edges) 
stand out more accordingly. However, since an optimal 
degree of emphasis depends on balance with a filter 
used in the normal process, weights are preferably fi- 



nally adjusted with reference to the output result. For 
example, the coefficients are preferably changed in cor- 
respondence with the number of effective pixels and 
their distances from the pixel of interest. 

5 [01 31 ] As described above, according to the third em- 
bodiment, when the filter coefficients are adjusted in cor- 
respondence with the number of lines that can be proc- 
essed at the same time even for image data to be ras- 
terized, a natural, continuous image processing result 

10 at boundaries of processing regions can be obtained. 
Hence, even when an image is placed obliquely, or is 
deformed into an arbitrary shape, it can undergo appro- 
priate simulate illustration process. 
[01 32] The third embodiment has explained the meth- 

15 od of converting RGB information of an image into lumi- 
nance/color difference information, filtering only the lu- 
minance information, and multiplying color difference 
components by a constant on the basis of the required 
brightness. Of course, RGB information itself or image 

20 data in other formats may be filtered. 

[0133] For example, original image data having a for- 
mat other than the RGB format may be tentatively con- 
verted into the RGB format, and the individual color 
components may undergo the simulate illustration proe- 
ms ess of this embodiment, or the original image data itself 
may undergo a filter process according to its data for- 
mat. For example, when original image data has the 
YMC format, the signs of all filter coefficients used in the 
third embodiment may be inverted. 

30 [01 34] Furthermore, the third embodiment can be ap- 
plied to original image data having the YHS format, 
L*a*b* format, and the like. That is, in such formats, 
since a luminance component can be easily extracted 
from an original image, only the luminance component 

35 can undergo the filter process, thus reducing the com- 
putation volume compared to filter processes in units of 
components of the RGB format. Especially, this effect 
becomes higher with increasing filter size. 

40 <Fourth Embodiment 

[0135] The fourth embodiment of the present inven- 
tion will be described below. Since the system arrange- 
ment in the fourth embodiment is the same as that in the 
45 first embodiment, a detailed description thereof will be 
omitted. 

[0136] Contrary to the need for higher-definition color 
image processes that pursue reality of photo images, 
the need for a monochrome edit process represented 

50 by a sepia tone process is still high. The monochrome 
edit process itself has prevailed. When an image that 
has undergone simulate illustration process by the filter 
process described in the first to third embodiments fur- 
ther undergoes monochrome edit, a unique image like 

55 a sketch by means of monochrome crayon can be pro- 
vided. 

[0137] In the fourth embodiment, the user can set an 
arbitrary hue upon designating the monochrome edit, 
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and when the user also designates simulate illustration 
process, the simulate illustration process is done prior 
to the monochrome edit. 

[0138] For example, the user sets desired image 
processes using a user interface (to be abbreviated as 
a Ul hereinafter) shown in Figs. 19A to 19C, and can 
preview the processing result. 

[01 39] Fig. 1 9A shows a default window of the image 
process setup UL After the user checks a check box of 
"Simulate illustration", as shown in Fig. 19B, he or she 
can adjust a "Contrast" adjustment lever for setting con- 
trast (brightness). In accordance with the adjusted po- 
sition of this lever, a preview image that has undergone 
simulate illustration process is displayed. 
[01 40] Fig. 1 9C shows an U I example set with "Mon- 
ochrome Effects", i.e., the monochrome edit, after the 
"Simulate illustration" process shown in Fig. 19B. In the 
fourth embodiment, upon setting the monochrome edit, 
the user can select a hue such as sepia or the like which 
is popularly used, and can also designate an arbitrary 
hue other than popular ones from the entire hue range 
using a color bar. Of course, by selecting only "Mono- 
chrome Effects", the monochrome edit using achromatic 
color can be done. According to Fig. 19C, yellowish 
green is set as user designated color, and a preview im- 
age after the monochrome edit using that designated 
color is displayed. 

[01 41] When the user is satisfied with the preview im- 
age on the Ul, he or she selects an "OK" button to close 
the detail setup window of the image process, and the 
setup contents are held in a RAM (not shown) or the like 
in a printer driver. In this manner, upon executing the 
image process, an image that has undergone the edit 
process the user desires can be obtained on the basis 
of the held setup contents. 

[0142] As has been explained in the third embodi- 
ment, when the simulate illustration process is done in 
units of luminance/color difference signals, the color dif- 
ference signals are set in an arbitrary hue, thus allowing 
easy and quick monochrome edit. For example, when 
an original image shown in Fig. 20A undergoes the sim- 
ulate illustration process and the monochrome edit proc- 
ess using sepia, a unique image shown in Fig. 20D can 
be obtained. 

[0143] As described above, according to the fourth 
embodiment, since photo image data can undergo the 
monochrome edit process using an arbitrary hue in ad- 
dition to the simulate illustration process, even the user 
who has poor knowledge about image processes can 
easily obtain an image that has undergone desired proc- 
esses including many patterns, unique image process 
effects, and the like. 

[0144] Hence, a photo image can undergo an edit 
process in higher degree of freedom, and an image with 
higher originality can be easily created. 
[0145] in the fourth embodiment, a single image un- 
dergoes the simulate illustration and monochrome edit 
processes as setups of the image processes. Of course, 



other image processes may be simultaneously execut- 
ed. For example, by setting the roughness of the edited 
image using the Ul, arbitrary resolution conversion can 
be achieved. 
5 ~~ 

<Other Embodiments> 

[01 46] Note that the present invention may be applied 
to either a system constituted by a plurality of devices 
10 (e.g., a host computer, an interface device, a reader, a 
printer, and the like), or an apparatus consisting of a sin- 
gle equipment (e.g., a copying machine, a facsimile ap- 
paratus, or the like). 

[0147] Concerns of the present invention can also be 
15 achieved by supplying a storage medium, which records 
a program code of a software program that can imple- 
ment the functions of the above-mentioned embodi- 
ments to the system or apparatus, and reading out and 
executing the program code stored in the storage me- 
20 dium by a computer (or a CPU or MPU) of the system 
or apparatus. 

[0148] In this case, the program code itself read out 
from the storage medium implements the functions of 
the above-mentioned embodiments, and the storage 
25 medium which stores the program code constitutes the 
present invention. 

[0149] As the storage medium for supplying the pro- 
gram code, for example, a floppy disk, hard disk, optical 
disk, magneto-optical disk, CD-ROM, CD-R, magnetic 
30 tape, nonvolatile memory card, ROM, and the like may 
be used. 

[0150] The functions of the above-mentioned embod- 
iments may be implemented not only by executing the 
readout program code by the computer but also by some 

35 or all of actual processing operations executed by an 
OS (operating system) running on the computer on the 
basis of an instruction of the program code. 
[01 51] The present invention includes a product, e.g., 
a printout, obtained by the image processing method of 

40 the present invention. 

[0152] Furthermore, the present invention also in- 
cludes a case where, after the program codes read from 
the storage medium are written in a function expansion 
card which is inserted into the computer or in a memory 

45 provided in a function expansion unit which is connected 
to the computer, CPU or the like contained in the func- 
tion expansion card or unit performs a part or entire 
process in accordance with designations of the program 
codes and realizes functions of the above embodi- 

50 ments. 

[0153] In a case where the present invention is ap- 
plied to the aforesaid storage medium, the storage me- 
dium stores program codes corresponding to the flow- 
charts (Figs.3, 4, 8, 9, 13, 14 and 17) described in the 
55 embodiments. 

[0154] In addition to JPEG used in the present inven- 
tion, various other methods (for example, a method 
called scale index coding) can be used as block coding. 
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[0155] As many apparently widely different embodi- 
ments of the present invention can be made without de- 
parting from the spirit and scope thereof, it is to be un- 
derstood that the invention is not limited to the specific 
embodiments thereof except as defined in the append- 
ed claims. 



Claims 

1. An image processing method comprising: 

the modify step of modifying a first image to 
obtain a second image on the basis of a first signal 
obtained by extracting an edge of the first image, 
and a second signal obtained by reducing the 
number of tone levels. 

2. The method according to claim 1 , wherein the sec- 
ond signal is a signal processed by decreasing the 
number of tone levels of the first image and increas- 
ing brightness. 

3. The method according to claim 1 , wherein the mod- 
ify step includes the step of simultaneously gener- 
ating the first and second signals as the second im- 
age. 

4. The method according to claim 3, wherein the mod- 
ify step includes the step of generating the first and 
second signals by filtering the first image using a 
predetermined filter. 

5. The method according to claim 4, wherein the pre- 
determined filter has coefficients, a sum total of 
which is positive. 

6. The method according to claim 5, wherein the pre- 
determined filter has coefficients, a sum total of 
which is 2. 

7. The method according to claim 5, wherein the pre- 
determined filter has a pixel of interest having a pos- 
itive coefficient, and all other pixels having negative 
coefficients. 

8. The method according to claim 4, wherein the pre- 
determined filter is a 5 pixel x 5 pixel filter. 

9. The method according to claim 8, wherein the pre- 
determined filter has a pixel of interest having a co- 
efficient = 26, and all other pixels having coefficients 
= -1. 

10. The method according to claim 4, wherein the pre- 
determined filter is a 3 pixel x 3 pixel filter. 

11. The method according to claim 4, wherein the pre- 
determined filter is defined by that some of coeffi- 



cients of the predetermined filter, for pixels other 
than a coefficient of a pixel of interest are "0". 

12. The method according to claim 4, wherein the mod- 
5 ify step includes the step of setting the filter on the 

basis of brightness of the first image. 

13. The method according to claim 12, wherein the 
modify step includes the step of setting a larger sum 

io total of coefficients in the filter as the first image is 
darker. 

14. The method according to claim 13, further compris- 
ing the luminance distribution calculation step of 

J5 calculating a luminance distribution of the first im- 
age, and 

wherein the modify step includes the step of 
detecting brightness of the first image on the basis 
of the luminance distribution. 

20 

15. The method according to claim 14, wherein the 
modify step includes the step of detecting bright- 
ness of the first image on the basis of an average 
luminance of the first image. 

25 

16. The method according to claim 14, wherein the lu- 
minance distribution calculation step includes the 
step of generating a luminance histogram of the first 
image. 

30 

17. The method according to claim 16, wherein the 
modify step includes the step of detecting bright- 
ness of the first image on the basis of a median of 
the luminance histogram. 

35 

18. The method according to claim 16, wherein the 
modify step includes the step of detecting bright- 
ness of the first image on the basis of a maximum 
frequency value of the luminance histogram-. 

40 

1 9. The method according to claim 1 4, further compris- 
ing the correction step of correcting the first image 
on the basis of the luminance distribution, and 

wherein the modify step includes the step of 
45 setting the filter on the basis of a correction condi- 
tion generated in the correction step. 

20. The method according to claim 4, wherein the mod- 
ify step includes the step of modifying the first image 

so in units of blocks each consisting of a predeter- 
mined number of lines, and controlling a size of the 
filter in accordance with an object line position for 
modifying in the block. 

55 21. The method according to claim 1, wherein the first 
image is a photo image. 

22. The method according to claim 21 , further compris- 
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ing the decoding step of decoding the first image if 
the first image is block-encoded. 

23. The method according to claim 22, wherein the 
modify step includes the step of modifying the first 
image decoded in the decoding step to obtain the 
second image after the first image is smoothed. 

24. The method according to claim 21, wherein the 
modify step includes the step of tentatively block- 
encoding and decoding the first image when the first 
image is not a block-encoded image, and then mod- 
ifying the first image to obtain the second image. 

25. The method according to claim 1, further compris- 
ing the output step of outputting the second image 
onto a recording medium. 

26. The method according to claim 1 , wherein the first 
signal is a signal obtained by extracting an edge 
having a thickness according to an attribute of the 
entire first image. 

27. The method according to claim 26, wherein the at- 
tribute of the entire first image is an image size. 

28. The method according to claim 26, wherein the at- 
tribute of the entire first image is an image resolu- 
tion. 

29. The method according to claim 26, wherein the at- 
tribute of the entire first image is manually set. 

30. The method according to claim 1, further compris- 
ing: 

the instruction input step of inputting a user in- 
struction that selects a desired one of a plurality 
of modify modes; and 

the image process step of executing an image 
process for image data using the modify mode 
corresponding to the user instruction, and 
wherein the plurality of modify modes include 
an illustration mode for converting the first im- 
age into the second image in the modify step. 

31. The method according to claim 30, wherein the in- 
struction input step includes the step of selecting 
the first image from a plurality of images, selecting 
the illustration mode as the modify mode for the first 
image, and setting detailed modify contents in the 
illustration mode. 

32. The method according to claim 31, wherein the 
modify step includes the step of setting a filter on 
the basis of the detailed setup contents in the in- 
struction input step, and generating the first and 
second signals by a filter process using the filter. 



33. The method according to claim 32, wherein the de- 
tailed setup contents include setups of brightness 
of the modified image. 

5 34. The method according to claim 33, wherein the 
modify step includes the step of setting a larger sum 
total of coefficients in the filter as brightness of the 
modified image set based on the detailed setup 
contents is higher. 

10 

35. The method according to claim 31 , wherein the de- 
tailed setup contents include setups as to whether 
or not the modified image is grained. 

'5 36. The method according to claim 35, wherein when 
the detailed setup contents are set with grainy con- 
version of the modified image, the first image is con- 
trolled to have block distortion. 

20 37. The method according to claim 36, wherein when 
the detailed setup contents are set with grainy con- 
version of the modified image, the first image is con- 
trolled to be a JPEG-encoded image. 

25 38. The method according to claim 30, wherein the plu- 
rality of modify modes further include a mono- 
chrome effect mode for converting the first image 
into a monochrome image. 

30 39. The method according to claim 38, wherein the im- 
age process step includes the step of executing the 
illustration mode for converting the first image into 
the second image and then executing the mono- 
chrome effect mode for converting the second im- 

35 age into a monochrome image when the user in- 
struction designates both the illustration mode and 
the monochrome effect mode. 

40. The method according to claim 39. wherein the il- 
40 lustration mode includes the step of converting lu- 
minance and color difference signals of the first im- 
age, and 

the monochrome effect mode includes the 
step of converting the color difference signals of the 
45 second image. 

41. The method according to claim 38. wherein the in- 
struction input step allows to set a hue in the mon- 
ochrome effect mode. 

50 

42. The method according to claim 1. further compris- 
ing: 

the segmentation step of segmenting the first 
55 image into a plurality of regions: and 

the setting step of setting processing contents 
of each of the segmented regions, and 
wherein the modify step includes the step of 
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modifying the first image in units of regions to 
obtain the second image. 

43. The method according to claim 42, wherein the seg- 
mentation step includes the step of obtaining a plu- 
rality of regions by generating the first image in units 
of regions. 



input means for inputting a first image; 
modify means for modifying the first image to 
obtain a second image on the basis of a first 
signal obtained by extracting an edge of the first 
image, and a second signal obtained by reduc- 
ing the number of tone levels; and 
output means for outputting the second image. 



44. The method according to claim 42, wherein the set- 
ting step includes the step of setting filters in units 
of regions, and 

the modify step includes the step of generat- 
ing the first and second signals by executing a filter 
process using the filters set in units of regions. 

45. The method according to claim 44, wherein the set- 
ting step includes the step of setting a filter in cor- 
respondence with a size of the region. 

46. The method according to claim 45, wherein the set- 
ting step includes the step of setting a filter in cor- 
respondence with the number of lines in the region. 

47. The method according to claim 44, wherein the filter 
set in the setting step includes a filter having differ- 
ent line and column sizes. 

48. The method according to claim 44, wherein the set- 
ting step includes the step of setting coefficients of 
the filter for each region. 

49. The method according to claim 48, wherein the filter 
has coefficients, a sum total of which is positive. 

50. The method according to claim 49, wherein the filter 
has a pixel of interest having a positive coefficient, 
and all other pixels having negative coefficients. 

51 . The method according to claim 50, wherein the filter 
increases absolute values of coefficients for pixels 
other than a coefficients of a pixel of interest as a 
size of the filter is smaller. 

52. The method according to claim 51 , wherein the filter 
has coefficients other than a coefficient of the pixel 
of interest, which are set to be powers of 2. 

53. The method according to claim 44, wherein the set- 
ting step includes the step of setting a predeter- 
mined filter for a region having a size not less than 
a predetermined size. 

54. The method according to claim 44, wherein the set- 
ting step includes the step of setting the filter on the 
basis of a user instruction. 

55. An image processing apparatus comprising: 



56. The apparatus according to claim 55, wherein said 
10 modify means obtains the second image by filtering 

the first image using a predetermined filter. 

57. The apparatus according to claim 55, wherein said 
output means prints out the second image on a re- 
's cording medium. 

58. The apparatus according to claim 55, wherein the 
first signal is a signal obtained by extracting an edge 
having a thickness according to an attribute of the 

20 entire first image. 

59. The apparatus according to claim 55, further com- 
prising: 

25 instruction input means for inputting a user in- 

struction that selects a desired one of a plurality 
of modify modes; and 

image process means for executing an image 
process for image data using the modify mode 
30 corresponding to the user instruction, and 

wherein the plurality of modify modes include 
an illustration mode for converting the first im- 
age into the second image by said modify 
means. 

35 

60. The apparatus according to claim 59, wherein the 
plurality of modify modes further include a mono- 
chrome effect mode for converting the first image 
into a monochrome image. 

40 

61 . The apparatus according to claim 60, wherein when 
the user instruction designates both the illustration 
mode and the monochrome effect mode, said im- 
age process means executes the illustration mode 

45 for converting the first image into the second image, 
and then executes the monochrome effect mode for 
converting the second image into a monochrome 
image. 

so 62. The apparatus according to claim 55, further com- 
prising: 

segmentation means for segmenting the first 
image into a plurality of regions; and 
55 setting means for setting processing contents 

of each of the segmented regions, and 
wherein said modify means modifies the first 
image in units of regions to obtain the second 
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image. 

63. The apparatus according to claim 62, wherein said 
segmentation means obtains a plurality of regions 
by generating the first image in units of regions. 

64. An image processing system which connects an im- 
age processing apparatus for generating a second 
image on the basis of a first image, and an image 
output apparatus for outputting the second image, 

said image processing apparatus having 
modify means for modifying the first image to obtain 
the second image on the basis of a first signal ob- 
tained by extracting an edge of the first image, and 
a second signal obtained by reducing the number 
of tone levels. 

65. The system according to claim 64, wherein said 
modify means obtains the second image by filtering 
the first image using a predetermined filter 

66. The system according to claim 65, wherein said 
modify means executes different filter processes in 
units of regions of the first image. 

67. The system according to claim 64, wherein said im- 
age output apparatus is a printer for printing out the 
second image on a recording medium. 

68. The system according to claim 64, further connect- 
ing an image input apparatus for inputting the first 
image to said image processing apparatus. 

69. The system according to claim 68, wherein said im- 
age input apparatus is a digital camera for generat- 
ing an image signal by sensing an object image. 

70. The system according to claim 68, wherein said im- 
age input apparatus is a scanner for generating an 
image signal by optically scanning a photo. 

71 . The system according to claim 64, wherein said im- 
age processing apparatus further has: 



when the user instruction designates both the 
illustration mode and the monochrome effect mode, 
said image process apparatus executes the illustra- 
tion mode for converting the first image into the sec- 
5 ond image, and then executes the monochrome ef- 
fect mode for converting the second image into a 
monochrome image. 

73. Storage medium storing a program code of an im- 
10 age process, said program code including at least: 

a code of the modify step of modifying a first 
image to obtain a second image on the basis of a 
first sig nal obtained by extracting an edge of the first 
image, and a second signal obtained by reducing 
15 the number of tone levels. 

74. Storage medium storing a program implementing 
the method according to any one of claims 1 to 54. 

20 75. Recording medium on which the second image 
modified by the image processing method accord- 
ing to any one of claims 1 to 54 is formed. 

76. Recording medium on which an image that has un- 
25 dergone processes for emphasizing an edge, re- 
ducing the number of tone levels, and increasing 
brightness is formed. 

77. A method according to claim 1 , wherein the first im- 
30 age is a color image. 

78. A method according to claim 1, wherein the modi- 
fying step includes the step of modifying the first im- 
age by each color component. 

35 



40 



instruction input means for inputting a user in- <5 
struction that selects a desired one of a plurality 
of modify modes; and 

image process means for executing an image 
process for image data using the modify mode 
corresponding to the user instruction, and 50 
the plurality of modify modes include an illus- 
tration mode for converting the first image into 
the second image by said edit means. 

72. The system according to claim 71 , wherein the plu- 55 
rality of modify modes further include a mono- 
chrome effect mode for converting the first image 
into a monochrome image, and 
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